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; The feasibility Of cocstruc^^^ scores 

. which »iir yield pretest measures ha ving^ail the properties required 
by the special regressicTi model is explorei as an alternative to the " 

, single pretest score usually used in studi^t seiectibn fbf Eieientary 
Secbndary Educatibp Act Title I cbmpensatbry education program^^^ ^ 

s^Jeadinq dat^r including Stanfcrd Achi scores, obtained'^^ 

^^CB^stsd^ts in grades 2 , 3 ^ and 6 in/ f our schbol districts are . 
analyze^yfroB a technical and^a practical perspective, fSathematics - 
data ob^ined from students in grades 2 and -3 in one ichbbi district 
a-« a^^b analyzed. AlthbughccmpbSites do not seem tb increase the 
acc^rittcy cf : estimating the nb-treatment /expectatibn in regressibh 
designs variables^ such as teacher ratings and placement in a reading 
series can be quite reliable. M prcce^^ siiple sums 

5f/two or three carefully selected variables has bbvibus practical 
advantages oyer elaborate score trans for matibnsi ft can yield pretest 
me^ special 
'regressicn model. If, however r the scaling of a variable or score 

/ inflation on a rating causes floor or; ceilinig effects in the 
coiiposite sccres> problems are create pro5ect gains are 

estimated with the speciai regression Bodel. (BL) . 



ERIC 



* Eeprcductions supplied by EDES are the best that ca^ .* 
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ft paper preseated at. the? ^ual meeti ftmerican. Educational 

Research &sso<^i^tic^> gbstoh^ Aptii 7--p-r; 1980.- 
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. .^ The evaiuation ofy^aucational ptogtams ;in fieia^-settirifs where trtie 
ramdOTization is: not possible allows eonaitions to be assbciiat^ with the 



tceatment ^ich may pcsW' rival eji^ilanatidtis" of the resuits- 



Quasi-experimertta^^^ points out ^ must have 

expi^rci't- models indbr|bratihg differences between the trea^iprxt and 



'groups over time. Since the selection of disadi 
> students for. sui^lemehtary instruction is by nature non-ranSjli^^e' 
^ Title 1 evaluation Sjrstem (Taiimadge s Woodr 19780 provides three 

alter native models foe estimating the impact of compensatory education 
programs^ This paper 'Will I focus on selection issues surrounding only brie 



of/ these r the special regression modelr sometimes referred 'tb,:aa'^ G. 
The special regression model may- be used 'in situations where 
r treatment students are identified -^lely bri the basis of the pretest, 
. ^The. model assumes* that the regressibri lirie oDmputed from the cbmparisbh 
group pre- and posttest scores m^ be used to predict^at the treaitoent 
group mean would have been without Title I instruction, / In other words ^ 

_ ■ ' • ■ ■ 

the comparison group regression 'line is projected bei^ a strict pretest 
• •■■* : _^ • ■ ' . \ , - ' * • , 

, cutoff tb predict what tte performance of the Title I grbi^ would have 

'been without su^ienentai instruction^ The difference between this "nb" 
. treaSent expectation" cuid the observed treatment .group' mean' represents 

.the gain in achievement : attributable to Title I iristructibri. 
, The model assumes that pretest scores are' liriearly related to . 

pattest sabres and that the treatmerit arid:cbmparisbri grbup regression 
. lines af,e parallel. These assumptibhs may ribt be met when there are 

flo<h or ceilirig Effects bri either the pretest or posttest (Estes s 

Ahdersbrir 1978). A moderately high cbrr elation between the pretest and ' 



posttest arid a reasbriably large sample of -students are recommended to 

• ■ . • ■• . 

ensure ah accurate evaiuatioSi 
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i -^ S<Aooi districts pve attempted 'to implement t$e^B^cia^^regressioii 
teel^i^tfi miied-^ ptiinary diff iralty h^beefi^ 

•; ^election. For pra'^'cal bt^ pblitic^^ distric.^ have- been ! ] 

^-unable to "main^ih aVstrict .pretest cutoff, under these conditions the 

-il' ' .' '* • " ■■ ■ ■ ■' - ■ 

.model is unders|ecified and will produce misleading, resuife. - ■ " ' 



typicalfy the district evaluatbr will use one of the following 



procedures." to .analyze the data: 



•U) - EScclude treatment |tudehts above the cutoff and oompatisort. ' 

^ ^ 5tudertts below jthe cutoff from the analysis. • 

* - * • . ■ ■ *>' ■ . ' . 

(2) . Include all students in the analysis, whether selected 

ai^rbpriately'br notw 

*' ' ' ■ , ■ ■ ^. > 

(3) Follow the analysis procedures for. the nb^ references model' 
^ ; :?fe6dei'ftT. ■ / - . 

' . ■ r . - ... ■ • : '■ . * 

All three analysis procedures are unsatisf ac^ryw If t^e variable 
^dded to the selection .process correlates with^sttest ^s^^ 
two procedures tend to iinder estimate pr^^^ The first. ' 

_ prbced^ure may be use^f hbwevetf when the n&faer oB students that must be 
eliminated is very small or wher? the addfed v^riab^^ dbrri^late 
with pbsttest scores,- for ex^l^, if -^tudents^ m reassigned* due to"* 

random scheduling proble^^^ ' The ki/d proc^^^ tends to IWr estimate * 
project effectj|veness due to. regression to the mean sihc^ the pretest was 
used^ in part, for seiectidill. * 

A fourth, procedi^e was pro^sed by Yap> E^tes and Hahsen (1^79) who 
argued that an 'unbiased estimate of the treatment effect, could be 
Obtained by droi^ihg all treatment >and OTmparison students in a band 
where there was averiap betWeeir the\ two groups on the pretest. This 
procedure, however, wiii of ten result in a loss ih^ precision since the 
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• . • •_ ^ _■ __ _^ ■ ■ ■■ . *i' ■ -s 

wide band required in most situations .would cause too mahy stude&aft ta^ -u.* »•*•; ,. 

. eliminated frdn the ianaJ^sis. The mote restricted range of scotb'^hS^ 

; . " : - these extrgne groups : would m ake the procedure more sensifciv^ to floar Sna .'^ j*- 

.' *. • ■_ ■; ■ •• • ■ ' '■' ■ ' i_ ;« 

.: ' h ; ceiling effects^ > - ^ ' ' . " > : . : -. "ir^^'^^ 

^ ^ ' ^ Ae^^lie re a^ a r to be no .^geeptafci^post hoc ^roceaur^^^^^ — ." 

. * . ' p --^ ■ - • . - > • _ ' ■ ■ . - ■ ^ . - - ' 

applying the speciaa^ regression m have not been " — - 

assigned to the tr^tment solely bh the basis of the pretest scores^ Sore. 
, ■ . acceptable selection measures can be cbiiatructeS, ; Several /^variables A 

- Which ,the di-strict wished to use for student selection era be dbmbinid to 
^^■> fbra 1978) • fts long as a striict' .cutoff on 

. : t^^ selection, the 'obmposite sdbres 

ay be used in the^ analysis, 5d construct the rompositer tfe variables 

■ J- ^ ^ ■ _ ■ . ■' • ■ - . . *■ . ■ 

are either standardized or simply muitipiied by constants so that ekch • 

has the same standard deviation . and thus the same conttibutioi^ to the 

- TOmpbpite. A. well constructed composite can increase the accurac^f ^"t^ 

regression model ind the validity- of ^the selection prbcedure, ' ■ 

' Despite the: technical rationale for using cbi^pbsite scores ^ district^ 

" - . ■ . * ' • .• 

, , af e skeptical. They want to Know if all this extra wbrk is worth; i£. - 

: ■ - . _ ^ ' ■ ' ■ ■ _ i: . ■ 1 .' , ' ■ ' ■ 

Will composites improve the accuracy bf MbdeljG? Will theJa^^ite tend 
to select' the studeiSis teachers think should be served? Can^simpie/ , 
composites -be designed that do not require excessive hand cotn|>utajtibh %t 
expensive aamputer equipment? This paper reviews the empirical findings 
-^f previous ^ttempts^ by districts to use composites and discusses the - 



er|c ' ■ * 



iinpiicatibhs of these findings for Title I student selection arid th^ 
special regression model. . : iW v 



PIEIiD ftK>iiiiaTI(^S OP COMPpSITE SCORES 



■{ Pour, ippiications Of composite scores by school districts are: ' - r ' *. 

reported vhere to exj^ore issues in the imfaemeritatibn of ccamposxte scores '^^ 

•for Title i student selection^ " The exan^les differ cbhsiderabiy in: the ' 

measuresi^eiected for the »mposite, the way the composite was ' . * * 

constructedr .and the' role the composite played in selection^ -: t^s:-' ' , 

-'■ _ ■■ :■ ■ , ■■■ ■ ■ - 

. natural, variation is useful for interpreting, the results of each ' ^ ' ' 

. • __ . ' ^ _ ■■: r- • ' ;V. r 

-district. It should be noted that each of the districts designed and. *' ^ 

- - ■ ./ > i; 

carried out the i>rocedures des^xbed below ^atlsout research or ^valuali^ri - 

st^tffs^ witfcut/OTmputer fTO^ and with iittle or no techriic^L ' 

assistance. - • * ; . 



-vDistridt A constructed a oompoisite. pretest measure consisting 6# 

--^ • _ _ _■ _ ,__ _ _ ^ . . _ • ■ ■ __ __J^ 

pretest scores and teacher ratings which was used for sisiecting stqdents 
- ■j_ - ■ ■ - . , ■ V - • 

and estipating project gains in a special regression model d^ign. The' 

. ; Stanf ord Achievene^ Test was administered, to grades 2 through 6 in the . 

— — - . 4 ■_ ■ ^ _ / 

spring. Classroon teachers were asked rtb rate the reading ability of 

' ■ . ■ - . - - . ._ . * ' ^ 

each student on a fi^^ point scale. fhe-iaSing t^ srored 10, 30^ SOr 

^ ^^9e of scores siS.lar to the pretest. S^:t^\:^ - 

. . . . ... - . . • . . ^ . ^ ■ • ^ 

. district wished- to weight pretest scores more highly than ratings r the- - 
composite score tos computed by the formula: _ * ' . * ' 

• e = i.4X + . ;;. " . 

where C is the individual's compbsit sdbre^ X is the pretest percentile, 
and R. is the teacher rating. : ._ * 
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\ ' The means, staridara- deviktions, ^an<^ cbtrelat^^^ for pretest^ rating; 

and - posttest are giv^eil ifi-^TaM-e^d; The rating correlates 
highl^r w.^th- the prWti^t^ff^^ 
~^^^es> .^^^iqb , score Sid : npt cbrrela|e signif idantiy higher with: 

=posttei5t^^TOres^1?fian^<3i^-^i 



It is interesting to libte that tfie scaling of the teacher 'ratihg^ . 
; produced a STOre d£stribu:^on that w^s d-ose to normal for eich grade. , 
\ _ Mearp^^ahd standard deviatiOTS for the r^ti^ were yery-'clbse to what one' 
^i^d\^ 5tf andvn^'p6r' respectively. Thus; tb^ rating 

As might be 'expected r the results of an analysis of covariahce ' 
-suggested that there were/ significant diff erehces-in the m^^ riti^rtgs by 
various teach^r^.^ s'c^Sres were -used is the covariate to partially 

. oDrrect for toe acixievemeht^ieve^ the class. The high obrreiatibns * 
observed above, ' though^ suggest that these iriter-ratb^ differences did. 
; not seriously af f et^t; the validity, of the rating. _ " " - ' 

. ^?^^fflTle__B . ^ ^ . ^ 

District B administer ed the Total Reading _^d Total Jlathematics • 
subscales of the Stanford Achiev^ent Test each spring to evaluate the' 
reading Md mathemati« progr^ at grades 2 and 3. Although the' 
district had been uSn§; evaluation in(Ddel, there was 

. seme eyid^c^ ttet toe .equipfercentile a^^ hot hbid for their 

•students; In the primary grades,. students cohsistehtiyperfo 
cxsmpared to the national average, but "the class average droped in eich 
successive year until it approached the national average at about grade, 
fivei^ ^ 
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.Since studeiitji w«e sel(BCted Using" teacher judgment, the"* datg were 
•.. •• not apptppriate f or analysis using th^, special regressittn iodei, but*' th^ 
district did collect ratings amd' bther ihf bnhatibh-%rOT . 
w^re oOT^ted. vmile ^thes^^ ^fbsi^s.W^^ availaBl« tot t^e teacher -when , 



^sttidents-were-assi'ghe^ tp^Tifcie I:»^i:tey were^na^ ased to - set a ^ery 
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unrestrictiW upp^r limit abb^re' Which ^studeii^^^ 

-. - - . ■ • -.^^ J^ :^ ._ _^ ■_ : !- 

..Title I serviced. TKus^ , the match. between -ionqposite scored arid teacher 

judgments a5uld l?«. e3%>lbred: up^^^ actual reifei^Sig oa^ed'i tb indicate 
_ title I iristf ucticin ah<3 .(J,.tp',indicate no sa^i&ental Instruction. 

the. composite wa% constructed during the .fall: . Classf obm - teachers 
were asked to rate each gtii^ent in ieventeen: r^a^ihg*, mathematics jahd"'' 
study sJcill areas on a f5jjf ^iht^cale. - Itt mat'hSnitics- an inform£^ i 
inventory, develop ^ the' district ,ras alsb .administered. ■ IS rea^Sg, 
Students were assigned 'points -on the basis', of the grade plac^ent of ^s 
or her current level in the reading series. Ah attempt -was made to 
equate the difficulty of levels of the ttefee different reading series 
used across buildings. Each element of the COTpositer^pretest, rating 
and informal iiweritbry or readitig placement; was then'rescaled / 
assigning points to each score so that the rescaled values oh each s<x)re 
ranged fran 0 to 15» The points weri simply totaled to f orm the ' . ' ^ 
ccaapbsite scbrew ^ 

Ohfortuhateiy the point sj^tem was devised without looking at score . . 
distributions,' and as a result, rather extrene floor and ceiling 
variables were observed on each of . the variables. Figuri 1 shows the 
score distriTsutions for each variable and the composite totals for grade * 
3 reading. By way of cbmparisbh* the original pretest distribution in ^ 
Normal Curve Equivalents ^SCEs) is also prodded, although the' raw : * ' 

SOTtes f or reading plaTOiaent -^d .t^^^ • 
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^-^^J^i* distributions probably roaghiy apprbxSat6 noCTaiity. The 

; -sdorte by a cwnstaint and roonding- to a whole number.^ An explanation for 
-jT • . /i->ith^ i^ yairiable was suggested district • 

officials'^.' ITeachefs probably had a/t?esponse set to overrate the needs' of 
stud€uits due to tt^ length of , £he rating scale and thus ensure that \ 

certain students TOuld hotV be ^ * 

■■■ • ' . ■ ■* ' '■-''*' * , 

• - The me^a&, stanSard deviaticsis and cbrr^atibhs' for each of the 

• . — -_- 1- ■ _ ___ __ 

-COTipo3ite and* cr iter lc«: variables are given in Table 2 for students in, - 
, reading.,- Correlations 'amohg the 'composite ^riables w^e moderate while 
•> correlations w^th the oampbsite were high. Correlations of this 
variables with the criterion variables were moderate to, high; 

— _ ■- _ _ * ■ *__ _ ^ -Sims - 

N The composite correlated highly with the posttest, but not 

sighi'ficahtly higher than the pretest alone." The composite wrr elated ^ . 

very highly with referrals, significantly high^^than the pretest alone. 

These results suggest that t^iie t^ use c»f compdsi^es would hot ihcrie 

the precision of the special regression model, a <ioser match to the . 

subjective selection procedure is possible. "V - . 

. inter-rator differences were agaaii observed" 'across teac^ but - 

the^e did not appear to seriously affect the results^ ' ^ 

Table 3 presents the means ^ standard deviations cofxeiations : 

among' the cbmpbsite and criterion vatiabi^s for students in matheitiatics. - 

The results were similar to those observed in. reading excapt^that- a£ 

grade 3 the composite did not correlate more highly with teachpr judgmeht 

-fthan the pretest alone. . _ 

r ■ ■ : - : . 

i Example C ^ " ^ ^ ^ ' « ^ ' * 



Wfth the belg.of an outside fentr actor ^ District C evali^ted'ats 
Title ! reading program in grades 1 through 6 using the special' •* ■. V- ' 
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regression model. Students were selected with a single scbife' cutoff on a 
of teacher- r atings ana scale d scbtes on the Total Reading ' ji^i; 



subscale of the*. Stanford Achievement Test. ; Each spring/ clissroan ' 
f • , teachers rated students -pn eight rjadihg and 'study skills With a five. ^ 



^int scale axid^ administered" the achievement test* The cOTposite' was 
instructed using the re<»mmended procedure for multipiying oiie of the; 
v^variabies by a constant to equate the standard deviations' and soraming^ the 
■■■ Spores.' - ■ 

^ Although it was only pc«sible to examine the scores from one 
el^erttary school^ the data from District C are: of special interest si-nce 
two iridependent ratings of same students by different teachers vrere 
' atrailable. . One -rating :was made near the aamiriistratibn of the pretest 
and the other, was made a year later. This prpv-ided 'the opportunity to 
Study the- consistency, of , ratom and to examine tlie eff 6c ts of string 'the 
two ratings^ . . " 

The means, standard deviations and obrrel^ions -for .the pretest, 
teacher ratings, . s unseed ^ratings ^ and pdsttest are presented in Table -4. ' 
Again the prrelatibh between ratings and ties t* scores are- hx^h. The 
composite correlates highly with the posttest but no higher than the ' 

• _ ^ * _ _ '■ ' ' • ' ■ ■ ■ ^ . ' ' . • . . ■ . 

pretest alone. v 

— - . • ' ' - ■ ' ' -> 

; The correlation between. the twb ratings is remarkably high 

cdpsidering that different teachers made- the ratings a year apart. When 

the. ratings were combined to increase reliability, higher correlations 

with test scores were observed but the sample was tfob saall detect 

significant differences. - • , ' -1 
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. The fourth c^^_^ai3^ be desctibe d, only br iefly -due to .the smiit r 
amount of data available and certain probleos in the prbce^ osedr 
Pretest 'soorm^^ teacheg rat i^^ahd select^ 



for seasnd grade students orily. .Ratings were obtained as part of a f 
district feasibility stu^ by iski^ the ^issrodm\teacher to .estimate 
each students', reading ability on a six point , scale. . The directions for 
; -the scale encouraged the teacher tb iake^ use of t'^t 'scored' or informal ; 
ia^eaitbries 'Jx> help make their' jad^ente.V It 'is no^ known whether ' 
eeachers actijally made use Of t^ 

'She means., standatrd deid.ations aisd dsr relations for the pretest^ ,; • : 
ratiijg and teacher referrails are presented ih.TaBle 5, The high • 
* correlation between- the . pretest and. rating is hot surprisihg^if teachers . ' 

' •* ^- '.vj. ■ - ; ■ . :^ ■ - -\' _ Z'^-. 

did; use iest scores as the basis of ^theip rating, v However > the; ratings ^ ■ 
^ «>rrelated significantly higher with ref errals th^^ did the ■ pretest. 
When a con^site.rwasv fonned ulihg .leMt 'squares regr^^ - 
ratings added; significantly to' the predictioa^of ^acher placement 
^decxsions,: These^5ii^n§S suggest that -the ratings reliably iaeSsur^ 
'^scme aspect of telacher jud^ente not c^pturedr by, test scores'. ' ' . 







. _ ■ -v ^ _■_ . ji". 

DETELQPiNG ebJffiOSilES FOR STDDENT iSELECTIdN 



The -iour example- of f ieid-appiicatiiapjs of compbS;ite: scores, described 



select Title . I students^ The T^riation. iiv tte way Mch aistrictv 



agprbachea tfie task provides sdnebtsis for 
clearly these dHa are not without limitatiohs. 



though 




The specif ic pr<^ used to select' 'it Sdents for Title^-I service 

vary considerably fron^ aistrict-tb district and" are usually • . 
unsystematic.. When; teacher ^ is the basis-^or Title I referrals/ 

one or tnore of the f qMpWihg' -class^^^ of variables generally eh*^r into « 
j-;the jdeciQion: " X: • . ^ *, 

^garformahce indicatbrs—lndirators of .achievement include tes t 
sx»resi -grades, infoCTal i-nv^itorie^^^ reading, 
. series, previous parti qtpaiits iii Title 1 and classroom pirfonnance;' 
Study sifttls-^ eachers generally ibbk for attentiveness , work' 
efficiency, ability to oDmplete assignments and a'bility .t<5 f bllb^ ' 
directions. . : 

• Non-acadCTic factors— Less directly related to achievement are such 
factors as motivatibn^ persbnaHty ^d be 

, S >■.«*. 

_ _ _ • ^ _ __ ^ ■ __ - ^ . * ■ ■ . ' ' ' 

\ Aaninistf ative prba^s— There mav be Scheduling conflicts^ 

bverlappihg services, parental concerns, br other 2tdiinistrative 
; p^bbl^ tStt affect the decisibri. ' '« 
. / ^^he task bf choosing a|^)Mpriat^ variables is not^Jtb be taken 
lightly- Clearly it would be impossible to include more .than a siall 
number of the vktiabl^s listed abby^^^ in a formal selection procedure. 
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with tb* exception of performance indicator; the Variables, would 
difficult to measure^ More i^rtant , the' ^lidlis^f many of these, 
variables with respect -to selecting the most "needy" students is 
. qaestionabWi -Current per foraance in the sl^^ 
• criterion of primary interest to mc»t dis other 

factors usually are considered as w^ll. 

Scores on standardized tests are generally a first choice for ^ 
inclusion in fa ODmpc^ite. ; They are reliaH.e and -ralid indicators. 
Teachers^ however r feel that test scores do not adequately measure a 
^ /student'^s for servi and that their easperience with students in 

the classrocm enetbles them to make, more valid placement of students. To 
^ 'quantify these. tfea^^ are frequently used as in the ' 

four cases d«cribed here, yet ratings are notoriously subjective and 
- unreliable (Thorndilce &'Haganv^l^^^ How can ratings be expected to 
contribute to selection? ShbulS other indicators be used? ^ ^- 
Te a cher ratings . Several different rating- scales were des«ibed in 
the last -jsectibnr_^f rem simple five-point scales to lengfihy skills - 
checklists. Despite the various procedures used, ^ ratings 6f 

student ability were shown to be star 

Cbrrelatibns with test srores, with teacher referrals (District B) , arid 
with independent ratings (District C) were consistently ia the range Of 
.6D to .75. Moderate correlations were oteer^^ vw.th reading 
and an informal* inventory in District B. These findings ar^e consistent 
with other field studies' (Hennessyf Takbta & Ames/ 1980*). 

With the^high reliability bf the pretest scores and the redundancy 
among all -the achievement indicators^ it is proba&y unreasonable, to 
e3cpect much' imprbvCTeht in predicting pbsttest scores by adding 



12 



12 
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:dM^sites. the other handy teacher ratings as seen tb .reiiabiy 
measure sane factor that is iii^ by teachers in seiectihg; studentsN^but ii 
• Thus by inciuding ratirifs^ the district 

•should be better able to hold to a single score cutoff on the pretest - ' 
, canpositei Teachers should be more satisfied with the referrals based on 
cx>in^5sites than pretest sdores alone . 

The inflated :tatings and ihter-rator differences observed by these . 
districts suggest that care in 'collecting ratings is necessary-; Clear v 
4instouctLots or^^trsi^ig should be prbvided sp that tlsachers^^& V 
how to ma^e the .rating/ minimizing inflated ratings and ensuring a hbrmal 
distributibh Of scores. Teachers should be told what percent Of' their " 
students cah.be expected to, fail under, each ratihg,. though. this will vary 
with gbod'and bad classes. It could be noted that inflated ratings 

- _ - ■ i' _ i - ■ . ■ ■ .. : 

reduce the variability or range of scores, thus reducing the cbhtributibn 
to the composite^ and that students with l^ss need may get selected over 
students wit& more need. 

Probably one of the more effective ways to increase the reliabilLity 
of a rating scale is to use more than one rater. Only a single 

elementary teacher will be familiar with a student's reading or math . 

*--■».-•. 
performahce during the school year > but: two ratings could be easily 

obtained if classroon teachers rated their students iri the spring and 

again a <x>uple weeks into the fall. While the teacher has had relatively 

little time to,bbserVe the stiMeiit in the fall, reasonable ratings can be ' 

made as- the District B results show. Treating the inter-rater ■ ■ 

correlations provided by District C as reliability estimates, we can 

predict the reliability expected from cOTbining the :two ratings using the 

Spearman Brown Prbs^iecy Formula. If the. reliability of ratings A and B . 

are .70, the reliability of the combined ratings would be a sizable .82. 

■13 ■ ■ :■. ' 



fts the results in Table 4 suggest^ the cdttflbined rating shbuia correlate 
nor e highly with test scores. 

In view of the high reliability of ratings in the exan^es described^ 
inter-rator differences do not s^^ to be a major prbblen* The 
procedure described ty Roberts and Tallmadge {1976) to correct, for th^si 
differences are rather ccmplicated to implement by the districts and, are 
probably unnecessary." The procedure des^ibed above for obtaining two 
ratings oh the same students would se^ to be preferable. 

Other perfbrmamce iridicatots ^ Besides -standardized t^st sbdres and ' - 

teacher tatings:^ a number of other , indicators of achievement were listed 

above that <x>uld be used in a composite. There would only be two reasons 

■' ^ ' . ■ . ■ . 

for including additional performance indicators: to increase the* 

reliability Of the ODmposite and thus the <»rrelatibh with pc»ttest^ 

scores or to increase the -validity bf seleoUbh. The efforts of District 

B to include ratingsr iii^ritbri^ and reading placement in cSposites had- 

little effect ch cor r el atibns with the pbsttest, Qn the other hand, this 

district had gbbd reasbri tb consider other indicators thought" to more 

accurately reflect teacher 3a^^ents in student selection. Grade 

placement in the reading series correlated more highly with referrals ' 

* ' .. . • . 

than did teacher ratings. 

Other factors . ■ Not surprisingly, each of the districts, focused only 
on performance measures. It could be argued that poor study skills are 
relevant to need but no data was available (with the exception of several 
itaiB on.District B's rating checklist). Administrative prbblens that 
are unrelated to level bf achievement would hot heed to be included $ince 
students Iffected dDuld simply be drbpped fron the evaluation. When 
forced to bhgectify the decisibh process; teachers omitted \^riables that 
might >have been included otherwise but weri difficult to defend. 

- ... : 14 ■ ■ 



Scj^Ttnfi th4 -Variabl^ ^ 
Once the ^rakiaEL ^ 



tve been 



. nmibers shbuia be 4^^^ ^ ^^^^ 

(b) the variables have tte same standard 



scxire distributions ' |; normal , 
deviatidh^ ^ ^nd ^ (c)| tfee ^^iahdes 
probl^s^nTOOTterf^ byj the fdtir 



to floor ^^d «i^ effects it 



phosen for the selection measure ^ 



are scaled in the same direction. The 
districts and the sbiutiqns to them' 



an interesting iperspec^^ on tfiese issues. . . , *" 
gCTre dtstribilitions . Since. tLe- special- regression model is sensitive 



is partricuiariy i^ i'SspSct the 



SOTre distribution^ ih.^ing thk variaH.es. The odd distributiors 



displayed, in ^igurj aj for examplle. could have been avoided quite Easily 
■ imd the assignment of | points ; refljected the full range of values on each' 



variable. ; The' <dxse approximations to. normality found for the other 
composites was Jatiher' encouraging, especially since one might have 
expected inflated] scores on the ratings by teacljers wishing to make sure 
stu^dnts -suf f icieht points to be selected. 

Stahdardiaatibn. From the practitioner's perspective, standardizing 
ae T^riabies^ or nultiplying by constants as recommended, (Roberts and 
Tallmadge, 1976) is a nuisance. All but one district tried to 
approximate standardization by setting up • equivalent scales on each 
variably. The rating scale' used by Qistrict A, for example, yielded 
means and standard d4»iations remarkably close to that of the pretest 
NCSs. The point system constructed by District B achieved the same goal 
bat with- much tedious converting of scores. 

The advantage of simple scales may be-»obvious when computet * 
facilities are not available, but even when they are, the turnaround time 
required to' generate the composites could delay the start of the -program 

and manual computation may have to be used anyway. 

. w ^ ' ' ■ , ' 15 
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Weighting the Compo s i te 

-'^ • •• _ ^ _ ■ __ ■ 

. Assigtiing. weights to each .variable is /the final step in oihstructing 



• V 



the coinpbsite*/ The weighti^^ scheme used^ though^ is ;\bf seconSary * 
imjortahce relative to variable selection and scaiingi A^The ^^iSitbr can 
assign arbitrary weights to give one variable a larger OTntribution in 
t^e composite or find the least squares regression ooeff ici|erits that ^ 
maximize the correlation between the canposite and sane ^iterioa using 
multi'^e regressions When the correlatifeis between the varia^es and ' . 
with the .criterion are all of about th^ same magnitudev though, Mt'is 
• unlikely that unequal weights will substantially improve on unit or equal 
weights (Wainer, 1976^. 

To illustrate this pointy unit: -weights and optimal least squares 
weights wer^ cxsipared for Districts A aiid The. differential weights^ 
applied ty District ft (i*4y .6} were al to. examined ^ Muitiple regression^ 
wa^ used to find the least squares regression ob^ficiehts with pbsttest 
scores as the cariterion, Sincej/ the least squares solution capitalizes on 
chance variation J n the data, the Darlington (1968) formula was used to 
estimate the shrinkage that wouid'f-be observed in the .multiple 
correlations if the^regressicai coefficients had been applied to a new _sgt 
of data in a cress-validation design* 

The squared cotreiatibhs reported iri Table 6 generally suf^rt tftle 
notion that equal weights work fine in most situations* There are pnly 
small differences between the weighting schenes. The optimal- weights are 
slightly higher than district or unit weights but this difference' 
disa^ears .when the squared correlations are adjusted for estimated 
bverf it to the data. 



1 7 ■ . 

^ ^ 16 ; • . 



The special; regression model requires that the treatment group be 
select^a strictly oxt the pretest measure. Schema: districte appi^ng th^> 
,mpdel to Title I ev$iluation have fbuhdt it difficult to mee^ this 
requirement when teacher judgments or other factors Were ad^ to the 
student selection process but hb£ to pretest' measure, TMs paper 
• explored the fe^irbility of U3ihg composite scores tp solve this problem, 
• : _ As .Dawes and Cbrri^^ make gcfed models 

of ; decision making behavior. When the appropriate variables are incTuded 
xn the-model^ OTm|»s scores make excellent predicSioriS/ of teache^^ 
-re^rcals for Title I instruction, .Wiile the referrals process ' cahSbt be 
- fflpdeled exactly , a. ^ppsite provides a better match' than test scores , 

J- •; _ ' . ' ' ■ ' _ .■ . ♦ A 

alone; and- sliouid make it easier for a district to ehf br<^ i -strict * 

cutoff. The dis.tl:ict must escamirie its selection prbre to determine hB5 

._ _ : ■ N ■ ■ - 

it can. best be modeiedf '^"^^"^ - v 

. Contrary to expectations (Roberts & Tallmadge > 1976) , composites' do 
not sem to increase the acciiracy of estimating the no trelttoeqt 
expectation in regressibi^Tfeigns; "^A^^ with^the data repDc ted here, : 

the pretest-ppsttest correlation was Sot increased by adding other . 
Jferfbrmfhc^ indicator^. Seastir^_^p^^^ t placement^ 
in a reading s^^^ cast be. quite rjeliable but/are redundant with pretest 



'Scores. I ^ 



^ :is|enOTdraeriftg to firu^ ; ^. 

•constructing composites, yield prete^ all the 

pro^rtira necessary for the special regression model. V ft, procedure 
reqciiring Simple sum of twp^ or ' three variables .has bbvibui practical 



advutages over elabdtate score transfbraatibns or scbijihf sc^ At 
least in the \atetern states^ few districts have the resburc»s' for 

"* ■ * ■ L » . . ' 

r - , . 1 . ■ * • . * . 

following the technical recdinmehdaidbris for designing o^pc^sites to the 
"letter* . ■ 

' Care in designing these single procedSes is required, though. If, 

for example r .th^ scaling of a variable or score inflation on a rating - 

~ - . ^ * ■ ■ - ' 

scale causes floor or filing -effects in the composite scores, problens 
are created when project ga^ns are to be estimated with the special ^ 
regression model. ' * * ^ ^ - _ 

- ^ Independent, of the e^lukt ion model used/ ev^uatbrs and program > 
.spe^alists should have a concern for the validity bf tt^ process 
which, students are selected for bbnipehsatbry educ^fcibh prbgr^ v 
Hermessy, Takatat and Ames (1980) suggest that many "needy" students are 
TOt beirfg served by Title 1. While the <def ihitioh of heed used in the 
study requires sane discussibiif those authors are hbt albhe in their 
assessment Of current selection procedures. Composite scores provide ah 
bb^ectivejr observcible 2d.tef native to the decision malSng processes 
^x^orily us^« 

As Dares iid eorri^n (1974)' put its 

. /.iin analysis^ of ^he tasks faced by the decisibh, maker leads tb the 
V conciasio^ thatj linear m^ ther^fbrer-hbt 
• . surprising thattiinear' mc^els putperfprm intuitive judgment. Nbr is 
it SQx^ising that -decis^Wi mak^s (ir^bf ar as they are. behaving 
'apgro^riately) are^^parMfeph^ by linear 

modeis...The whole trick is_tb decide" what variiables to look at and 
then' know how. to add. (p. 105) ' • 
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Figure 1. Score distributibris bh cbmpbsite variables and composite 
:total for third grade students in Reading from District B (N = 168); 



Keahs/ Stahdaurd Deviation and Correlations 
for District A Students in Reading 



Grade 2 (N » 897) 

(1) Pretest 1 

(2) Rating 

(3) Ccipbsite 

(4) Pbsttest 



X 

55.0 
43.4 

109.1 

r 

55;7 ^ 



19.7 



21.2 
50.2 
S0.2 



.75 
.98 

,75 



Cogreiat ions 



3^. 



.84 
.65 



.76 



^rade .4j (M 5« 942) 



. (1) 


Pretest! 


5^.2 


-J (2) 


Rating 


49.^ 


(3) 


Ccmfbsite 


111.4 








(4) 


Pcsttest 


,51.4 



17 . 8 
*20.7 
47; 1 
19.2 



.76 

.98 
.70 



.86 
.66 



.73 




51-6 


17.9 




51.9 


■22. 


.74 


104-3 ; 


:47.6 


•97 


; 51^5 


15.4 


.75 



.86 



.78 



^Pretest and^Mttest means oh scores are. reported in Normai Curve 
Equivalents (NCEs) . 
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_ Table 2 J 1 : 

: _ j!^aMr Standara pe^at4o^ erid Cbrteiktibris 

ftoong Ccmposite yariabies /_ Pbsttest and. Treatiii^t f 
District B Stuaehts ^iri Reading . . . 







\, V 








t 


SD 


X 


^ 


_ _^ ^ 


4 


5 • 


% 


Gratje 2 120) 


















11) PretMtl 


» - » 

8.3 


...6,3 




< 


■* 








(2) Reading level ^ 


'9,9 


5.4 


m i *k 












V (3) Checklist 


10.4 


■ 4.6 






■V' 








' (4) Composite 


28.4 


14.3 




.92* 


.82 


i ■ 






(5) ;j?osttest2 - 


' 59; 9 


.20.9 


— go 

: ' ' ^ 






- 

_ V' ' 
' ^*84 




^ \ V 
^* 


{61 Referrals 


• .62 


<49 


.70 




. .61 ' 


• OO * 


GradeV3,_(N =*,97) 






• 








r ■ 




(1) Pretest _ 


9.3 


5.8" 










■-i - 




(2) ; Reading Level 


9.3 


; - 6.3 


.68^ 

* 












(3} Checklist - • 


9.3 ■ 


5.2. 


.58 


.46 










(4) CfHnposite 


27.8, 


14.7 


.88 - 


,85 


.79 








(5) Pd6ttest2 . 


54^9 


18.4 


-.79 . 


-.63 


-.49 


-.76 






;(6) Referrals 


.65 


' .48 


. 77 ' 


.79 ■ 


.64' 


. 86 " 


t 

-.66' 




t ■ it , . 










.* 








• ■ • - f 















^^Pretest scor« were obnverted. to a 15-pbiht scale. 

^egatiTO-^xreiitions reflect the reversed pealing of the oomjos-ite 
variables. ' : ^ - • 
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- : ' ^. • - ; _: * _ Table 3 :^ 
.Means y Stahaajrd'Tpey^^^ 

:* Ccmjp^^ite.Variaaa and Trea^ent for"\ 

^ • : \- • bistrict - B gtuaehts in^Mat'fiSiati^ f ; ' 











: r- - 






. - . — Correla>tl( 












.... * 


















g) •- 


■ 1 


2 


3 




, . 5 


• 


Grade 2 (N* » 104h • 
. (1) Pret^tl :. 


8.2 




• 






• I* ' 


^ " • » 

♦ ■• • • -.N • 




V <2) inventory . 


: 7.0- 


. 4.8 


.6tt J 












. (3) . Rafei^t' - 








> ; .45' 










( 4 ) Camposi te ^ 5 : 


25, 9 - . 


•12.7 




.83;\: 


- • , 

• .:73-'- 










59.3 


f*20.8 


-.76 


•-.51 


->-,44 


-.75 




'* - . 


' • ■ * ■ •" " \ ' ' ' r 

- • — - -■' '*■ ' — ' 












; y . 


* (6) 'Referrals * 


.62 


■ ,49 


75. ' 


. . . 75. 


.' .55 


.84 ' 


-.58;. 




Grade .3- (if = -96) t i ^ - > 
- (1) PretJtl ; ; 


.... ^ 


. -6.3 










• ■ .■ 




A (2) Inventory- r 


12.2 . 


: 3.7 


. -64 












(3); Rating 


■ 9i3*. 


• -5.3 


,54 




> 






« 


* - ' " y*- ' • • ' * ^ , 
\.' t(4J 'Composite::;* 


30.3 


12.9 


.88 ; 

; . ^ 




. . .83 






■J*.' 


1 ^ • ' i-j '. ■- . ■ • — " 
(SK Po^^est? 


58.2 


20. 9 . 


''-.72>- 


-.56 


-.49 : 


^.70 






OS) Referrals 


.69 


.46 




.85 


.65 




f 83 


-.64 



*^l^retes4t scores wire .oonverte^iifib. a 



■ 1 - :: 



Negative cotr^atiors ' refiecfi- fc^V rev^^ SCTliiig of the cbm^site 
variables^ . * ■ " * ^ ^ ^ ' - , - . 

.-^tr- .■' ' V ■ • ^- g -.-r ■ - % .'-V ' 
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_ ; _ _ :: Table 4 
Means^ Standard Pe5n.atibhs and Cbrrelations for 
--.Composite Variables and Postt^t for District C 
Students iii 











^TCoEteiatibns " 






' ■ - 


X 


Stf 


1 






1 -A 










2 


•a 






A- - • 


Gra<fe 2 (N i 46}^ 




















(1) Pret^tl 


,121.9 


15.0 
















(2) 'Rating ft 


20.3 


5.1 














\ 


13)^ Compbiite 


242. § 


34,3 


• 85 


. 84 












f4) POTttesJbl 


137^6 


12.8- 




7P 


- T*5 
„. . • 










(5) Rating 


24.5 


5.3 


.68 


' .78 : 


' ill 










(6) A + B 


44.8 


9.8 


■ .76 


' .94 


.82 


.80 


' 95--' -' 






Grade 4. (N > 45 




i 
















•^■(i) Pretest i 


142.9 


t7i5 










* 






(2> Rating A 


23.2 


6.7 


■ .45 ■; 






■ • " 






\ 


(3) Composite 
^ (4) Pbsttesti 


f 285.7 

153 ;i 


29.8 
18.2 


.75 


76 
.54 


" ■ 7'i .1 


- '. ■ ■ 








(5) Rating. B ; 


24.6 


8.7 


. .77 


.72 1 




7ft 




• 




(6} A + B ' 


47.8 


14.3 


.68 


.90 




.72 


-95' 




















■ . • ■ . 


Srade 6 (N » 44) • 














1 






(1). Pretest! 


158.3 - 


18.4 - 








\ 








(2) Rating A 


22.1 




.69 






- 








(3) Ccmposite 


317. 0 


.^34.9; 


.92 


■ .87 












(4) PcMttesti 


167.6 . 


16.8 


.82 • 


.71 


.81 










(5) Rating B 


23.3 


8.1' 


.75 


.76 


.80-: 


.71 ' 








^'■^ (6), A + B 


45,5 


13.8 . 


.77 . 


.93 




.80 


• 95 
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Pretest and pbsttast are reported in expanded ^ scale Scor^wv 



. ■ table* 5 ; ; ' 

■ Heaii8> St«aara *: 
aaor^ Pretest SdDrea^ Ratings Teacher Referra^^ 
for iSistrict D Students in Reading {N i 242); 







SB 




.atlotts — 


■ 1 


— ^ — 


(1) Pretest -.^ 

(2) ' Rat^l 

(3) . Refferalsl ; 


57.8 

' 3.7 

.28 ;:. 


2i;o' 

. 1.4 
.45 


■ : -.54 '■ . 


-.63 ' 



negati;TO vOT the f eversed scaling 
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- . ; Table 6 _ 

Squared Cotrelatidns Betweerr tSe fittest 
aiid Three Weighted Composites 



Grade 



N 



District A, Reading 
2 897 

4 . ■. 942 / 
6 '. 855 • ■ : 



District 



\^ Weights 



.581 
.534 
.626 



Unit 



.558 
.528 
.628 



OptimeU. " 
Weights . 



.582 
.535 
.631 



izdSptiipal^ 

tfeights 
(Adjusted) 



.580 
.533 
.629 



District B, Reading 
2 -> 120 ' 
3- 97 



.533 
.558 



.543 
,629 



.515 
.600 



District B, Mathematii^ 

2 : 104 ' 

3 ^6 ' ■ / 



.571 
.496 



.605 
,541 



.577 
.505 
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